Image sensor and image sensing method

ABSTRACT

An image sensor and an image sensing method are provided. The image sensor may restore a high resolution image with respect to a high magnification based on sensing information corresponding to different fields of view (FOVs) and that is received through lens elements having a same focal length and different lens sizes.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. application Ser. No.16/394,822, filed Apr. 25, 2019, which claims priority from U.S.Provisional Application No. 62/741,147, filed on Oct. 4, 2018, in theU.S. Patent and Trademark Office, and Korean Patent Application No.10-2018-0154741, filed on Dec. 4, 2018, in the Korean IntellectualProperty Office, the disclosures of which are incorporated herein byreference in their entireties.

BACKGROUND 1. Field

Example embodiments consistent with the present disclosure relate totechnology for sensing an image.

2. Description of the Related Art

Due to development of optical technologies and image processingtechnologies, image capturing apparatuses are being utilized in a widerange of fields, for example, multimedia content, security andrecognition. For example, an image capturing apparatus may be mounted ina mobile device, a camera, a vehicle or a computer, to capture an image,to recognize an object or to acquire data to control a device. A size ofan image capturing apparatus may be determined based on, for example, asize of a lens, a focal length of a lens or a size of a sensor. Forexample, the size of the image capturing apparatus may depend on a sizeof a lens or a size of a sensor. As the size of the sensor decreases, anamount of light incident on the sensor may decrease. Accordingly, aresolution of an image may decrease, or it may be difficult to capturean image in a low illuminance environment. To reduce the size of theimage capturing apparatus, a multi-lens including small lenses may beused. When the size of the lens decreases, a focal length of the lensmay decrease. Thus, the size of the image capturing apparatus may bereduced based on the multi-lens.

SUMMARY

One or more example embodiments may address at least the above problemsand/or disadvantages and other disadvantages not described above. Also,the example embodiments are not required to overcome the disadvantagesdescribed above, and an example embodiment may not overcome any of theproblems described above.

One or more example embodiments provide an image sensor and an imagesensing method.

In accordance with an aspect of the disclosure, an image sensor includesa lens array including a plurality of lens groups, each of the pluralityof lens groups including at least one lens element, and a sensing arrayincluding a plurality of sensing elements spaced apart from the lensarray and configured to sense light passing through the lens array,wherein a lens size of one lens group from among the plurality of lensgroups is different from a lens size of another lens group from amongthe plurality of lens groups, and wherein a number of sensing elementsarranged along one axis of the sensing array is determined based on anumber of lenses for each of the plurality of lens groups.

The number of sensing elements arranged along one axis of the sensingarray may be determined based on a least common multiple of the numberof lenses for each of the plurality of lens groups.

The number of lenses for each of the plurality of lens groups may bedetermined based on a geometric progression.

The number of lenses for each of the plurality of lens groups may bedetermined based on an arithmetic progression.

The number of lenses for each of the plurality of lens groups may bedetermined based on a progression from which at least one prime numberin an arithmetic progression is excluded.

The number of lenses for each of the plurality of lens groups may bedetermined based on a progression that is based on a number of lensesincluded in a lens group corresponding to a default zoom.

Each lens group from among the plurality of lens groups may correspondto a respective lens size such that for each lens group from among theplurality of lens groups, lens elements included in the lens group havethe corresponding lens size.

For each lens group from among the plurality of lens groups, lenselements included in the lens group may be located adjacent to eachother.

One lens group from among the plurality of lens groups may include asingle lens element.

At least one lens element from among the plurality of lens elements maybe arranged to cover less than an entire portion of at least one sensingelement from among the plurality of sensing elements.

A processor may be configured to restore an image based on sensinginformation sensed by the plurality of sensing elements so that aresolution of a central region within a field of view (FOV) of the lensarray is higher than a resolution of a region adjacent to the centralregion.

The image sensor may further include a processor configured to acquire acompound eye vision (CEV) image based on sensing information sensed bythe plurality of sensing elements.

The processor may be further configured to rearrange pixels included inthe CEV image based on light field (LF) information sensed by theplurality of sensing elements.

The processor may be further configured to restore a scene image fromthe CEV image based on a geometric relationship between the plurality ofsensing elements and the plurality of lens elements.

The processor may be further configured to restore a scene image fromthe CEV image based on an image restoration model that is completelytrained before the CEV image is acquired.

The image sensor may further include a processor configured to selecttarget sensing information from among sensing information sensed by theplurality of sensing elements, the target sensing informationcorresponding to a zoom factor designated by a user, wherein theprocessor is further configured to restore a scene image based on theselected target sensing information.

The processor may be further configured to select, as the target sensinginformation, information corresponding to a field of view correspondingto the designated zoom factor.

Each lens element from among the plurality of lens elements may beconfigured to refract incident light and to form a focal point of lightexiting the lens element at a point on a sensing array including theplurality of sensing elements.

In accordance with an aspect of the disclosure, an image sensing methodincludes sensing, by a sensing array including a plurality of sensingelements, light passing through a lens array including a plurality oflens groups, each including at least one lens element, and restoring, bya processor, a scene image based on an intensity of the light sensed bythe sensing array, wherein a lens size of one lens group from among theplurality of lens groups is different from a lens size of another lensgroup from among the plurality of lens groups, and wherein a number ofsensing elements arranged along one axis of the sensing array isdetermined based on a number of lenses for each of the plurality of lensgroups.

In accordance with an aspect of the disclosure, a camera includes a lensarray including a plurality of lens groups, each of the plurality oflens groups including at least one lens element, and a sensing arrayincluding a plurality of sensing elements spaced apart from the lensarray and configured to sense light passing through the lens array,wherein a lens size of one lens group from among the plurality of lensgroups is different from a lens size of another lens group from amongthe plurality of lens groups, and wherein a number of sensing elementsarranged along one axis of the sensing array is determined based on anumber of lenses for each of the plurality of lens groups.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or other aspects will be more apparent from the followingdescription of example embodiments taken in conjunction with theaccompanying drawings, in which:

FIG. 1 is a diagram illustrating a structure of an image sensoraccording to an example embodiment;

FIG. 2 is a diagram illustrating an example in which a sensing elementreceives a ray through a lens element according to an exampleembodiment;

FIG. 3 is a diagram illustrating a relationship between a number ofsensing elements and a number of lens elements according to an exampleembodiment;

FIG. 4 is a diagram illustrating a number of lens elements and a size ofa lens element that are determined based on a focal length and a desiredfield of view (FOV) in a lens array according to an example embodiment;

FIGS. 5A and 5B illustrate an arrangement of lens elements in a lensarray according to an example embodiment and FOVs of a scene sensed bysensing elements through lens elements arranged as described above withreference to FIG. 5A;

FIGS. 6A, 6B and 6C illustrate examples of an arrangement of lenselements in which overlapping of sensing points is minimized accordingto an example embodiment;

FIGS. 7 through 10 are diagrams illustrating examples of an arrangementof lens elements according to example embodiments;

FIG. 11 is a block diagram illustrating a structure of an image sensoraccording to an example embodiment; and

FIGS. 12 and 13 are diagrams illustrating examples of an apparatus inwhich an image sensor is implemented according to an example embodiment.

DETAILED DESCRIPTION

Reference will now be made in detail to example embodiments which areillustrated in the accompanying drawings, wherein like referencenumerals refer to the like elements throughout. Example embodiments aredescribed below in order to explain the present disclosure by referringto the figures.

The following structural or functional descriptions merely describe theexample embodiments, and the scope of the example embodiments is notlimited to the descriptions provided in the present specification.Various changes and modifications can be made thereto by those ofordinary skill in the art.

It should be further understood that the terms “comprises” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, components or acombination thereof, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof. Expressions such as “at least oneof,” when preceding a list of elements, modify the entire list ofelements and do not modify the individual elements of the list. Forexample, the expression, “at least one of a, b, and c,” should beunderstood as including only a, only b, only c, both a and b, both a andc, both b and c, or all of a, b, and c.

Unless otherwise defined herein, all terms used herein includingtechnical or scientific terms have the same meanings as those generallyunderstood by one of ordinary skill in the art. Terms defined in generaldictionaries should be construed to have meanings matching contextualmeanings in the related art and are not to be construed as an ideal orexcessively formal meaning unless otherwise defined herein.

Regarding the reference numerals assigned to the elements in thedrawings, it should be noted that the same elements will be designatedby the same reference numerals, wherever possible, even though they areshown in different drawings.

FIG. 1 is a diagram illustrating a structure of an image sensoraccording to an example embodiment.

A quality of an image captured and restored by an image sensor 100 maybe determined based on a number of sensing elements included in asensing array 120 and an amount of light incident on a sensing element.For example, a resolution of the image may be determined based on thenumber of the sensing elements included in the sensing array 120, and asensitivity of the image may be determined based on the amount of lightincident on the sensing element. The amount of light incident on thesensing element may be determined based on a size of the sensingelement. When the size of the sensing element increases, the amount oflight incident on the sensing element and a dynamic range of the sensingarray 120 may increase. On the other hand, a resolution of an imagecaptured by the image sensor 100 may increase as the number of thesensing elements included in the sensing array 120 increases. Also, asthe size of the sensing element increases, the image sensor 100 mayimprove the quality of high sensitivity imaging at a low illuminance.

A size (volume) of the image sensor 100 may be determined based on afocal length f of a lens element 111. For example, the size of the imagesensor 100 may be determined based on a gap between the lens element 111and the sensing array 120. Because the sensing array 120 needs to belocated at the focal length f of the lens element 111 to collect light190 refracted by the lens element 111, the sensing array 120 and thelens element 111 included in the image sensor 100 may need to be spacedapart from each other by a distance equal to the focal length f of thelens element 111. The focal length f of the lens element 111 may bedetermined based on a field of view (FOV) of the image sensor 100 and asize of the lens element 111, for example, a radius of an aperture ofthe lens element 111. For example, when the FOV is fixed, the focallength f may increase in proportion to the size of the lens element 111.Also, the size of the lens element 111 may be determined based on a sizeof the sensing array 120. For example, to capture an image within apredetermined FOV, the size of the lens element 111 may need to increaseas the size of the sensing array 120 increases.

As described above, to increase a sensitivity of an image whilemaintaining a FOV and a resolution of the image, the size of the imagesensor 100 may be increased. For example, to increase a sensitivity ofan image while maintaining a resolution of the image, a size of eachsensing element included in the sensing array 120 may need to increasewhile maintaining the same number of sensing elements. Thus, the size ofthe sensing array 120 may increase. In this example, to maintain theFOV, the size of the lens element 111 may increase as the size of thesensing array 120 increases, and the focal length f of the lens element111 may increase. Thus, the size of the image sensor 100 may increase.

To reduce the size of the image sensor 100, design schemes for reducinga size of a sensing element while maintaining the resolution of thesensing array 120, or for reducing the resolution of the sensing array120 while maintaining the size of the sensing element may be used. In anexample, when the size of the sensing element is reduced whilemaintaining the resolution of the sensing array 120, the size of thesensing array 120 and the focal length f of the lens element 111 maydecrease, which may lead to a decrease in the size of the image sensor100. However, in this example, a sensitivity of the image may alsodecrease, and a quality of an image under low illuminance may bereduced. In another example, when the resolution of the sensing array120 is reduced while maintaining the size of the sensing element, thesize of the sensing array 120 and the focal length f of the lens element111 may decrease, which may lead to a decrease in the size of the imagesensor 100. However, in this example, a resolution of an image may alsodecrease.

Referring to FIG. 1, the image sensor 100 may include a lens array 110and the sensing array 120. The lens array 110 may include lens elements111, and the sensing array 120 may include sensing elements 121.

When a size of each of the lens elements included in the lens array 110decreases, that is, when a number of lens elements per unit area of thelens array 110 increases, the focal length f of the lens element 111 andthe thickness of the image sensor 100 may decrease. In this example, theimage sensor 100 may restore a high resolution image by rearranging andcombining low resolution images passing through individual lens elements111. Thus, a thinner camera may be implemented by increasing the numberof lens elements in the lens array 110.

As shown in FIG. 1, lens element 111 of the lens array 110 may cover apredetermined sensing region 125 of the sensing array 120 correspondingto a size of the lens element 111. The sensing region 125 covered by thelens element 111 may be determined based on the size of the lens element111. The sensing region 125 may refer to a region on the sensing array120 that is reached by rays within a predetermined FOV by passingthrough the lens element 111. A size of the sensing region 125 may berepresented as a distance D from a central point of the sensing region125 to an outermost point of the sensing region 125. The light 190passing through the lens element 111 may be incident on sensing elementsof the sensing array 120 included in the sensing region 125. The light190 may include a plurality of rays. A ray 191 may correspond to a flow(i.e., a path) of a photon 101.

For example, a lens size may correspond to a diameter of the lens.

Each of the sensing elements in the sensing array 120 may generatesensing information based on incident rays 191 passing through the lenselements 111 included in the lens array 110. For example, a sensingelement 121 may generate sensing information based on the ray 191incident through the lens element 111. The image sensor 100 maydetermine intensity information corresponding to an original signalassociated with points included in a FOV of the image sensor 100 basedon the sensing information output from the sensing array 120. Also, theimage sensor 100 may restore a captured image based on the determinedintensity information.

The sensing element 121 may also include a color filter to sense anarbitrary color. In this case, the sensing element 121 may generate acolor value corresponding to a predetermined color as sensinginformation. Each of the sensing elements included in the sensing array120 may be located to sense a color different from that of a neighboringsensing element that is spatially adjacent to the sensing element.However, an arrangement of the sensing element 121 and the color filterof the sensing element 121 are not limited thereto.

When a diversity of sensing information is sufficiently secured and whena full rank relationship between the sensing information and originalsignal information corresponding to the points in the FOV of the imagesensor 100 is formed, a captured image corresponding to a maximumresolution of the sensing array 120 may be derived. The diversity ofsensing information may be secured based on parameters of the imagesensor 100, for example, a number of lens elements included in the lensarray 110 or a number of sensing elements included in the sensing array120.

Although not shown in FIG. 1, the image sensor 100 may include a memoryconfigured to store an image restoration model used to restore an image,and a processor configured to restore an image using the imagerestoration model stored in the memory.

The image sensor 100 may restore an image robustly against various noisecomponents that deviate from ideal conditions. Also, the image sensor100 may restore an image without being affected by an arbitrary patternsuch as a Bayer pattern that may be arranged on the sensing elements.Examples of an operation of restoring a high resolution image based on aplurality of low resolution images captured using multiple lenses by theimage sensor 100 will be described below. The plurality of lowresolution images that are used in combination to form a high resolutionimage may collectively be referred to as a compound eye vision (CEV)image.

FIG. 2 is a diagram illustrating an example in which sensing elementsreceives rays through lens elements according to an example embodiment.

As described above, a sensing array 220 may receive rays correspondingto individual points 230 shown as X1 through X10. The rays may bedetected by the sensing array 220 through a lens array 210. For example,a plurality of rays may be emitted from each of the individual points230. Rays emitted from the same point may form a light field (LF). An LFemitted from an arbitrary point on a subject may be a field thatindicates both the intensities and the directions of rays reflected fromthe arbitrary point on the subject. For example, rays emitted from afirst point X1 may form a first LF, and may be incident on a firstsensing element S1, a fourth sensing element S4 and a seventh sensingelement S7. Rays emitted from each of the other points, that is, pointsX2 through X10 may also form a corresponding LF. The individual points230 may be, for example, points on an arbitrary object that may be asubject of imaging. Rays emitted from the individual points 230 may be,for example, rays obtained when sunlight is reflected by the surface ofan object. Although the lens array 210 includes three lens elements andthe sensing array 220 includes ten sensing elements (for example,sensing elements S1 through S10) as shown in FIG. 2 for convenience ofdescription, example embodiments are not limited thereto.

The sensing elements S1 through S10 may sense rays that pass through aplurality of lens elements and that overlap each other. For example, inthe lens array 210 of FIG. 2, a focal length between a lens element andthe sensing array 220 may be reduced. Thus, the first sensing element S1may generate sensing information, for example, an intensity value, thatis the sum of intensity values of rays that are radiated from the pointsX1 through X3 and that overlap each other. Thus, information generatedby each sensing element includes original signal information frommultiple points 230. An image sensor may restore the original signalinformation from the sensing information using a model as describedbelow, for example.

The sensing information generated by the sensing elements S1 through S10may be modeled to be original signal information, for example, anintensity value, corresponding to a ray incident from each of the points230, as shown in Equation 1 below.S=T·X  [Equation 1]

In Equation 1, S denotes a matrix representing sensing information, forexample, a detected intensity value, sensed by each of sensing elements.X denotes a matrix representing an original signal value, for example, acolor value of an incident ray, corresponding to a ray incident on thesensing elements S1 through S10 from each point. T denotes atransformation matrix that represents a relationship between sensinginformation detected by the sensing elements S1 through S10 and originalsignal information corresponding to an incident ray. Rays correspondingto the individual points X1 through X10, the lenses, and the sensingelements S1 through S10 in the structure of FIG. 2 may be modeled asshown in Equation 2 below. In Equation 2, the individual points X1through X10 may be modeled to be located at an infinite focal point fromthe image sensor. For example, a distance between the image sensor andeach of the individual points X1 through X10 may be greater than athreshold distance.

$\begin{bmatrix}{S\; 1} \\{S\; 2} \\{S\; 3} \\{S\; 4} \\{S\; 5} \\{S\; 6} \\{S\; 7} \\{S\; 8} \\{S\; 9} \\{S\; 10}\end{bmatrix} = {\begin{bmatrix}1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 \\1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 \\0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 \\1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 \\0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1\end{bmatrix} \cdot \begin{bmatrix}{X\; 1} \\{X\; 2} \\{X\; 3} \\{X\; 4} \\{X\; 5} \\{X\; 6} \\{X\; 7} \\{X\; 8} \\{X\; 9} \\{X\; 10}\end{bmatrix}}$

For convenience of description in Equation 2, X1 through X10 denoteoriginal signal information, for example, a ray intensity value, of aray corresponding to each of the individual points X1 through X10. Also,S1 through S10 denote sensing information, for example, a sensingintensity value, detected by the sensing elements S1 through S10. Forexample, a relationship, for example, the above-described transformationmatrix, between the sensing information, for example, color information,corresponding to the sensing elements S1 through S10 included in thesensing array 220 and an original signal corresponding to rays incidentfrom the individual points X1 through X10 may be determined based on anarrangement of a lens element and a sensing element, a number of lenselements included in the lens array 210, or a number of sensing elementsS1 through S10 included in the sensing array 220. In the example shownin FIG. 2, the transformation matrix T is determined based on therebeing ten sensors S1 through S10 and three lens elements.

An example in which each of the individual points X1 through X10 islocated at an infinite focal point from the image sensor is describedwith reference to Equation 2. For example, when each of the individualpoints X1 through X10 is located at a finite focal point from the imagesensor, the original signal sensed by each sensing element may changebased on a distance between a subject and the image sensor and ageometric structure of the image sensor.

FIG. 3 is a diagram illustrating a relationship between a number ofsensing elements and a number of lens elements according to an exampleembodiment.

At least one of a plurality of lens elements may be arrangedeccentrically with respect to at least one of a plurality of sensingelements. In other words, at least one lens element may cover less thanan entire portion of at least one sensing element. For example, theplurality of lens elements may be arranged eccentrically, and may not bein a one-to-one relationship with the plurality of sensing elements. Alens element may cover a non-integer number of sensing elements, insteadof covering an integer number of sensing elements. In an example, amulti-lens array structure may be implemented as a fractional alignmentstructure. Parameters of the multi-lens array structure may include, forexample, a number P of sensing elements and a number L of lens elements.A ratio P/L between the number P of sensing elements and the number L oflens elements may be determined as a non-integer rational number. Eachof the lens elements may cover the same number of sensing elements P/Las a pixel offset. In FIG. 3, ten sensing elements and three lenselements are shown. Therefore, each of the lens elements covers 10/3, orapproximately 3.33 sensing elements.

As described above, an image sensor may have an arrangement of lenselements that have slightly different optical center axes (OCAs) thanthose of sensing elements in a sensing array. Thus, lens elementsincluded in a lens array may receive different LF information. Since adirection of a chief ray of each of the lens elements is also changed,the image sensor may optically acquire a larger amount of sensinginformation. Thus, the image sensor may restore an image with a higherresolution based on a variety of sensing information acquired asdescribed above.

When lens elements included in a lens array all have the same lens size,a number of lens elements in the lens array and a number of sensingelements included in the sensing array may satisfy Equation 3 as shownbelow. In this example, the number of lens elements and the number ofsensing elements may be relatively prime.R=P=L×

+1  [Equation 3]

In Equation 3, R denotes an integer indicating a resolution of a sensingarray with respect to one axis (i.e., the number of sensing elements ina single row of the sensing array), P denotes an integer indicating anumber of sensing elements in the sensing array with respect to oneaxis, L denotes an integer indicating a number of lens elements in alens array with respect to one axis (i.e., the number of lens elementsin a single row of the lens array), and N denotes an arbitrary naturalnumber to satisfy Equation 3. In Equation 3, R, P, and L may berepresented on a two-dimensional (2D) plane. For example, the resolutionR may be a resolution of the image sensor with respect to a horizontalaxis or a vertical axis. The resolution R with respect to the horizontalaxis or the vertical axis may correspond to the number P of sensingelements in a single row in a direction corresponding to the horizontalor vertical axis. In this example, L may indicate a number of lenselements based on the corresponding axis. Thus, a full two-dimensionalresolution of the sensing array may be represented by R×R, a totalnumber of sensing elements in the two-dimensional sensing array may beP×P, and a total number of lens elements in the two-dimensional lensarray may be L×L.

For example, FIG. 3 illustrates a cross-sectional view of a horizontalaxis or a vertical axis of the image sensor. With respect to one axis,the number L of lens elements may be 3 and the number P of sensingelements may be 10, to satisfy a relatively prime relationship. In thisexample, approximately 3.33 sensing elements may be covered by each lenselement with respect to the axis.

In FIG. 3, a first lens element may cover all of sensing elements S₁through S₃ and ⅓ of sensing element S₄. A second lens element may coverthe remaining ⅔ of sensing element S₄, all of sensing elements S₅ andS₆, and ⅔ of sensing element S₇. Similarly, a last lens element maycover the remaining ⅓ of sensing element S₇ and all of sensing elementsS₈ through S₁₀. In other words, each of the lens elements may cover aportion of at least one sensing element corresponding to the samedisparity as 1/L (i.e., a portion of a sensing element that is aninteger multiple of 1/L) where L is the number of lenses.

Based on the above-described geometric structure of the lens array andthe sensing array, LF information from the combination of points 230sensed by a sensing element covered by one lens element may be differentfrom LF information from the combination of points 230 sensed by asensing element covered by another lens element. LF information mayrefer to information about a combination of a plurality of LFs from aplurality of points 230. For example, a first sensing element S1 maysense LF information with a combination of a first LF of the first pointX1 of FIG. 2, a second LF of the second point X2 and a third LF of thethird point X3. On the other hand, a second sensing element S2 adjacentto the first sensing element S1 may sense LF information with acombination of a fourth LF, a fifth LF and a sixth LF in the structureof FIG. 2. Thus, one sensing element may sense LF information differentfrom LF information sensed by another sensing element.

The image sensor may rearrange positions of pixels of a captured imagebased on a correlation between LF information. For example, the imagesensor may rearrange pixels of a captured image, for example, a CEVimage, to generate an output image in which pixels of sensing elementsthat sense similar LF information among the plurality of sensingelements are adjacent to each other.

The image sensor may rearrange pixels indicating an intensity value of asignal sensed by an individual sensing element, based on a similaritybetween LF sensed by the sensing element and LF information sensed byanother sensing element, as shown in FIG. 3. For example, it may bedetermined that a similarity between the LF information of two sensingelements increases as the number of overlapping LFs in LF informationsensed by the two sensing elements increases.

When points reflecting rays are assumed to be at an infinite focal pointfrom the image sensor, the image sensor may determine the LF informationsensed by each of the sensing elements. For example, the image sensormay determine points that emit an LF sensed by each of the plurality ofsensing elements based on a position relationship between the pluralityof sensing elements and rays radiated from points spaced apart from theimage sensor by a distance greater than a threshold distance. A pointspaced apart from the image sensor by the distance greater than thethreshold distance may also be referred to as a “point at an infinitefocal point.” The image sensor may rearrange pixels such that pixelsrepresenting points that are spatially adjacent to each other in asubject of imaging are adjacent to each other in the output image.

The individual points X1 through X10 of FIG. 2 are adjacent to eachother in an order shown in FIG. 2. For example, the first point X1 maybe adjacent to the second point X2, and the second point X2 may beadjacent to the first point X1 and the third point X3. Two adjacentpoints may be, for example, points that are spatially adjacent to eachother in a subject.

Both LF information sensed by the first sensing element S1 and LFinformation sensed by an eighth sensing element S8 among the sensingelements 311 that are not rearranged may include LFs from both of thesecond point X2 and the third point X3. Thus, the first sensing elementS1 and the eighth sensing element S8 may be determined to sense similarLF information. When pixels corresponding to similar LF information arerearranged to be adjacent to each other, Equation 2 may be representedas shown in Equation 4 below.

$\begin{matrix}{\begin{bmatrix}{S\; 1} \\{S\; 8} \\{S\; 5} \\{S\; 2} \\{S\; 9} \\{S\; 6} \\{S\; 3} \\{S\; 10} \\{S\; 7} \\{S\; 4}\end{bmatrix} = {\begin{bmatrix}1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 \\1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 \\1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\end{bmatrix} \cdot \begin{bmatrix}{X\; 1} \\{X\; 2} \\{X\; 3} \\{X\; 4} \\{X\; 5} \\{X\; 6} \\{X\; 7} \\{X\; 8} \\{X\; 9} \\{X\; 10}\end{bmatrix}}} & \left\lbrack {{Equation}\mspace{14mu} 4} \right\rbrack\end{matrix}$

Referring to FIG. 3, sensing elements 312 are rearranged based onEquation 4. In the actual sensing array, first sensing element S1 may becovered by the first lens, the eighth sensing element S8 may be coveredby the third lens, and a fifth sensing element S5 may be covered by thesecond lens. After rearrangement of pixels based on Equation 4, thethree pixels corresponding to first sensing element S1, eighth sensingelement S8, and fifth sensing element S5 are arranged to be adjacent toeach other.

The image sensor may apply a rearrangement and restoration to an imagecaptured by the lens array and the sensing array that satisfy arelatively prime condition and Equation 3, to restore a high resolutionscene image with colors more similar to the original scene.

FIG. 4 is a diagram illustrating a number of lens elements and a size ofa lens element that are determined based on a focal length and a desiredFOV in a lens array according to an example embodiment.

An image sensor may include a lens array 410 and a sensing array 420, asdescribed above. The lens array 410 may include a plurality of lenselements. A lens size of at least one of the plurality of lens elementsmay be different from that of another lens element.

For example, the plurality of lens elements may be classified into aplurality of lens groups based on a lens size. The lens groups may begroups of lens elements classified by the lens size. For example, lenselements included in the same lens group may have the same lens size,and lens elements included in different lens groups may have differentlens sizes. In other words, the lens size corresponding to any one lensgroup may be different from the lens sizes corresponding to all otherlens groups. The image sensor may include lens elements with variousFOVs. A FOV of a lens element may be represented as shown in Equation 5below.

$\begin{matrix}{{FOV} = {2 \cdot {\arctan\left( \frac{D}{F} \right)}}} & \left\lbrack {{Equation}\mspace{14mu} 5} \right\rbrack\end{matrix}$

In Equation 5, FOV denotes a FOV, D denotes a size of a sensing regioncovered by the lens element, and F denotes a focal length 419.

The image sensor may be implemented in (i.e., included in) a mobileterminal, and the focal length 419 may be limited to satisfy a formfactor of the mobile terminal. For example, the focal length 419 may beless than a thickness of the mobile terminal. To implement the imagesensor in the mobile terminal, lens groups may be designed such thatthey all have the same focal length 419. Additionally, the lens groupsmay all have different FOVs from one another.

To acquire information of a distant subject, the image sensor mayinclude the lens array 410 including lens groups with various FOVs. Forexample, when the focal length 419 is fixed for all lens elements asdescribed above, and when FOV of Equation 5 changes, a size (forexample, D of Equation 5) of a sensing region covered by a lens elementmay change. For example, the lens array 410 may be designed so that anumber of lens elements included in the lens group increases with anincrease of a zoom factor of the lens group. In this example, based on adesign of the lens array 410, the relationship between FOV and the sizeof the sensing region enables a constant focal length of the lens groupto be maintained. Also, a lens size of lens elements included in thelens group may decrease as a zoom factor of the lens group increases. Alens size of each of lens elements included in the lens group maydecrease as the number of the lens elements included in the lens groupincreases. Thus, a number of lens elements included in an arbitrary lensgroup may be determined based on a lens size, which may itself bedetermined based on a focal length and a FOV of the lens group.

Also, for each lens group i, a number of lens elements shown along oneaxis of the lens array 410 (i.e., a number of lens elements in each rowof the lens group) may satisfy Equation 6 as shown below.R _(i) =L _(i)×

_(i)+1 s.t. i=1, . . . ,M  [Equation 6]

In Equation 6, R_(i) denotes a resolution of an image sensed by an i-thlens group (i.e., a number of sensing elements in each row of a sensingregion corresponding to the lens group), L_(i) denotes a number of lenselements corresponding to the i-th lens group lens along one axis of thelens array 410, N_(i) denotes an arbitrary natural number, and M denotesa number of lens groups included in the lens array 410. R_(i), L_(i),N_(i), and M may each be an integer greater than or equal to 1. Also, imay be an integer greater than or equal to 1 and less than or equal toM. For example, a number of i-th lens elements included in the i-th lensgroup may be L_(i)×L_(i).

FIG. 4 is a cross-sectional view of the lens array 410 in which thenumber M of lens groups is 3. For convenience of description, a firstlens group 411, a second lens group 412, and a third lens group 413 areshown sequentially from the left. L₁ may represent a number of lenselements corresponding to the first lens group 411 along one axis of thelens array 410, and may be 3, as shown in FIG. 4. L₂ may represent anumber of lens elements corresponding to the second lens group 412 alongone axis of the lens array 410, and may be 9, as shown in FIG. 4. L₃ mayrepresent a number of lens elements corresponding to the third lensgroup 413 along one axis of the lens array 410, and may be 27, as shownin FIG. 4. Since the cross-sectional view based on one axis with respectto the image sensor is also shown in FIG. 4 as described above withreference to FIG. 3, L₁, L₂, and L₃ may each be a number of lenselements along the axis. The first lens group 411 may include nine firstlens elements obtained by 3×3 (i.e., three rows of lens elements eachincluding three lens elements), the second lens group 412 may include 81second lens elements obtained by 9×9 (i.e., nine rows of lens elementseach including nine lens elements), and the third lens group 413 mayinclude 729 third lens elements obtained by 27×27 (i.e., 27 rows of lenselements each including 27 lens elements).

The first lens group 411 may transfer rays within a first FOV 451, forexample, 77 degrees, to a first sensing element. The first sensingelement may be covered by a first lens element included in the firstlens group 411, and may receive light passing through the first lenselement. The first sensing element may sense information correspondingto a first zoom factor, for example, a zoom factor of 1×. The secondlens group 412 may transfer rays within a second FOV 452, for example,30 degrees, to a second sensing element. The second sensing element maybe covered by a second lens element included in the second lens group412, and may receive light passing through the second lens element. Thesecond sensing element may sense information corresponding to a secondzoom factor, for example, a zoom factor of 3×. The third lens group 413may transfer rays within a third FOV 453, for example, 10 degrees, to athird sensing element. The third sensing element may be covered by athird lens element included in the third lens group 413, and may receivelight passing through the third lens element. The third sensing elementmay sense information corresponding to a third zoom factor, for example,a zoom factor of 9×.

For example, the image sensor may be implemented as a single sensor, ora plurality of sensors that each include the same sensing elements. Inthis example, the image sensor may be designed so that each lens groupmay support the same resolution. For example, a sensing region coveredby any one of the lens groups may include the same number of sensingelements as that of another lens group. In this example, a number oflens elements R shown along one axis of the lens array 410 may satisfyEquations 7, 8 and 9 shown below.R=L _(M)×

_(M)+1  [Equation 7]L _(M)=α_(i) ×L _(i) s.t. i=1, . . . ,M−1  [Equation 8]R=L _(M)×

_(M)+1=L _(i)×

_(i)+1 s.t.

_(i)=α_(i)×

_(M)  [Equation 9]

In Equations 7 through 9, R denotes a resolution corresponding to oneaxis of a lens array shared by all lens groups. L_(M) denotes a numberof lens elements that correspond to an M-th lens group along one axis ofthe lens array. L_(i) denotes a number of lens elements that correspondto an i-th lens group along one axis of the lens array. N_(M), N_(i),and α_(i) each denote an arbitrary natural number. A number of lenselements included in each of a plurality of lens groups may bedetermined based on a resolution shared by the plurality of lenselements. Thus, based on Equation 9, which is derived by substitutingthe expression for L_(M) of Equation 8 into the expression for R ofEquation 7, each of the lens groups may have a resolution shared by allthe lens groups. Further, each of the lens groups may include a numberof lens elements which is relatively prime to a number of sensingelements covered by each of the lens elements.

Thus, the image sensor may apply a rearrangement and restoration to animage captured by the lens array 410 and the sensing array 420 thatsatisfy a relatively prime condition and Equation 9, to restore a highresolution scene image with colors more similar to the original scene.

For example, the image sensor may rearrange an image acquired by acamera that uses lens elements arranged in a Bayer pattern and a colorsensor (for example, a sensor to sense an image of a red channel, agreen channel and a blue channel in a visible light band) in whichsensing elements are arranged in a Bayer pattern, in an order of similarLFs, as described above. The image sensor may obtain a uniform colorpattern, to provide a color interpolation with a higher performance.

FIGS. 5A and 5B illustrate an arrangement of lens elements in a lensarray according to an example embodiment and FOVs of a scene sensed bysensing elements through lens elements arranged as described above withreference to FIG. 5A.

Lens elements included in the same lens group may be arranged to beadjacent to each other. In FIG. 5A, first lens elements of a first lensgroup 511 with a first lens size may be arranged adjacent to each other.Lens elements included in each of the other lens groups, for example, asecond lens group 512, a third lens group 513, and a fourth lens group514, may also be arranged adjacent to each other.

FIG. 5A illustrates a top view of a structure in which a lens array 510includes four lens groups, that is, M=4. The first lens group 511 mayinclude four lens elements arranged in a 2×2 shape, as an example ofL₁=2. The second lens group 512 may include 16 lens elements arranged ina 4×4 shape, as an example of L₂=4. Similarly, the third lens group 513may include 64 lens elements, and the fourth lens group 514 may include256 lens elements. Although an image sensor is implemented as a singlesensor for convenience of description in FIG. 5A, example embodimentsare not limited thereto. For example, the image sensor may beimplemented as four individual image sensors, each including one of theabove-noted lens groups, or as different types of image sensors, forexample, sensors having individual sensing elements that are differentin size.

For example, a processor of the image sensor may acquire a CEV imagebased on sensing information sensed by a plurality of sensing elements.In the present disclosure, a CEV image may refer to an image acquired byoverlappingly capturing the same or a similar scene, in a manner similarto the operation of the compound eyes of an insect. For example, theimage sensor may acquire a CEV image based on intensities of raysreceived from a plurality of sensing elements through a plurality oflenses arranged in an array.

The processor may generate a CEV image that includes 2×2 first groupimages obtained by sensing light received through the first lens group511, 4×4 second group images obtained by sensing light received throughthe second lens group 512, 8×8 third group images obtained by sensinglight received through the third lens group 513, and 16×16 fourth groupimages obtained by sensing light received through the fourth lens group514. The processor may rearrange the CEV image based on LF informationsensed by the plurality of sensing elements, as described above withreference to FIG. 3.

Sensing information associated with a scene and acquired through thelens array 510 of FIG. 5A is described below with reference to FIG. 5B.

FIG. 5B is a diagram illustrating FOVs of a scene sensed by sensingelements through lens elements arranged as described above withreference to FIG. 5A.

A lens array may receive light from the outside of an image sensor andmay transfer the light to a sensing array. Lens elements in the lensarray may have different FOVs based on a lens size. For example,referring to FIG. 5A, a first lens element included in the first lensgroup 511 may have a first FOV, a second lens element included in thesecond lens group 512 may have a second FOV, a third lens elementincluded in the third lens group 513 may have a third FOV, and a fourthlens element included in the fourth lens group 514 may have a fourthFOV.

Sensing elements included in a sensing region covered by each lens groupmay sense rays corresponding to a FOV of a corresponding lens group.FIG. 5B illustrates sensing information sensed by a sensing elementcovered by a corresponding lens group, for each FOV of a lens group.

A first sensing element covered by the first lens element may receiverays corresponding to the first FOV through the first lens element, andmay sense first sensing information 591 b. The first sensing information591 b may be sensing information associated with a region 591 acorresponding to the first FOV. A second sensing element may receiverays corresponding to the second FOV through the second lens element,and may sense second sensing information 592 b. The second sensinginformation 592 b may be sensing information associated with a region592 a corresponding to the second FOV. As shown in FIG. 5B, the region592 a may be smaller than the region 591 a. A third sensing element mayreceive rays corresponding to the third FOV through the third lenselement, and may sense third sensing information 593 b. The thirdsensing information 593 b may be sensing information associated with aregion 593 a corresponding to the third FOV, which is shown in FIG. 5Bas smaller than the region 592 a. Also, a fourth sensing element mayreceive rays corresponding to the fourth FOV through the fourth lenselement, and may sense fourth sensing information 594 b. The fourthsensing information 594 b may be sensing information associated with aregion 594 a corresponding to the fourth FOV, which is shown in FIG. 5Bas smaller than the third region 593 a.

As shown in FIG. 5B, since sensing information is collected only by thefirst sensing element, in an edge portion of the region 591 a outsidethe region 592 a corresponding to the entire FOV (for example, the firstFOV), a density of the collected sensing information may be less thanthat of a central region (for example, the region 594 a corresponding tothe fourth FOV). All sensing information sensed by the first sensingelement through the fourth sensing element may be collected in thecentral region, and thus a density of the sensing information may behigh relative to the edge portion. Thus, a processor of the image sensormay restore an image based on sensing information sensed by theplurality of sensing elements, so that a central region within a FOVcorresponding to the lens array may have a resolution higher than aregion adjacent to the central region.

In response to an input of a zoom factor from a user, the image sensormay generate a scene image that is zoomed-in based on the input zoomfactor. For example, the image sensor may select target sensinginformation corresponding to a zoom factor designated by a user fromsensing information sensed by the plurality of sensing elements. Theimage sensor may select information corresponding to a FOV correspondingto a zoom factor as target sensing information from the sensinginformation. The image sensor may restore a scene image using theselected target information. For example, when a zoom factorcorresponding to the second FOV designated to the second lens group inFIG. 5B is input, the image sensor may restore the scene image based onsensing information sensed in the region 592 a corresponding to thesecond FOV. An operation of restoring a scene image may include anoperation of rearranging pixels based on a geometric structure of theimage sensor, or an operation of estimating an output image from a CEVimage using an image restoration model.

When a zoomed-in image (for example, an image for which a FOV decreasesas the zoom factor increases) is restored, regions corresponding to FOVsprovided by each lens element for the zoomed-in image may overlap. Dueto the above overlapping of the regions, the image sensor may restorethe zoomed-in scene image with higher resolution by simultaneously usingall information from all lens groups, for example, using informationfrom lens elements having various lens sizes.

In FIG. 5B, rays reflected or emitted from points at an infinite focalpoint are assumed to be received. The sensing information of FIG. 5B mayinclude a point (hereinafter, referred to as a “sensing point”) at whicha point on the object at an infinite focal length is sensed. As shown inFIG. 5A, a density of sensing points may increase when the sensingpoints move from an outer edge of the FOV toward a central region. Whenrays are received from a finite focal point, a range of a FOV of eachlens group of the image sensor may change.

FIGS. 6A, 6B and 6C illustrate examples of an arrangement of lenselements in which overlapping of sensing points is minimized accordingto an example embodiment.

FIG. 6A illustrates an example in which a number of lenses arrangedalong one axis for each lens group is based on an arithmeticprogression.

A number of sensing elements arranged along one axis of the sensingarray may be determined based on the number of lenses for each of theplurality of lens groups. For example, a number of lenses in each of theplurality of lens groups and a number of sensing elements arranged alongone axis may satisfy Equation 6 described above. The number of sensingelements may correspond to a resolution. Although the example in whichthe number of lenses arranged along one axis of each lens group isdetermined based on a geometric progression to satisfy Equation 9 hasbeen described above in FIG. 5A, example embodiments are not limitedthereto.

For example, a number of sensing elements arranged along one axis may bedetermined based on a least common multiple of the number of lenses foreach of the plurality of lens groups, as shown in Equation 10 below.R=L _(LCM)×

_(LCM)+1, s.t. L=LCM(L ₁ , . . . ,L _(M))  [Equation 10]

In Equation 10, L_(LCM) denotes a least common multiple of a number oflenses included in each of the plurality of lens groups along one axis,N_(LCM) denotes an arbitrary natural number, LCM( ) denotes a functionindicating a least common multiple, and L₁ through L_(M) each denote anumber of lenses included in each of a first lens group through an M-thlens group along one axis. When Equations 6 and 10 are combined,Equation 11 may be obtained as shown below.R=L _(LCM)×

_(LCM)+1=L _(i)×

_(i)+1 s.t.

_(i)=α_(i)×

_(LCM) ,L _(LCM)=α_(i) ×L _(i)  [Equation 11]

In Equation 11, LLCM denotes a multiple of Li, and accordingly αi is anatural number. Ni is a product of two natural numbers, and as a result,all the lens groups satisfy Equation 6 described above. When a number ofsensing elements arranged along one axis in a sensing array isdetermined based on a number of lenses on one axis for each of theplurality of lens groups according to Equation 11, position overlappingof sensing points for a scene may be minimized. Thus, a wider variety ofinformation may be acquired without overlapping, and accordingly theimage sensor may sense image information that enables restoration of ahigher-resolution image.

Since a least common multiple condition needs to be satisfied, a numberof lenses arranged along one axis may be determined based on anarithmetic progression as shown in FIG. 6A. A structure in which anumber of lenses on one axis for each of the plurality of lens groups isbased on an arithmetic progression may be referred to as an “arithmeticprogression structure”. In FIG. 6A, a first lens group 611 a, a secondlens group 612 a, a third lens group 613 a, a fourth lens group 614 a, afifth lens group 615 a, a sixth lens group 616 a, a seventh lens group617 a, an eighth lens group 618 a, and a ninth lens group 619 a mayinclude two lenses, three lenses, four lenses, five lenses, six lenses,seven lenses, eight lenses, nine lenses, and ten lenses, respectively,along one axis. In this example, the least common multiple LLCM of 2, 3,4, 5, 6, 7, 8, 9 and 10 may be “2520”.

When a number of lenses on one axis in each of the plurality of lensgroups in the image sensor is determined based on an arithmeticprogression, the image sensor may more effectively restore consecutivezoom images. In an example, when a number of lenses on one axis in eachof the plurality of lens groups is determined based on a geometricprogression as shown in FIG. 5A, zoom factors may be set in an order of1×, 2×, 4× and 8×, and accordingly a portion of zoom factors may beabsent. In another example, when a number of lenses on one axis in eachof the plurality of lens groups is determined based on an arithmeticprogression as shown in FIG. 6A, a zoom factor may be set in units of0.5×, and accordingly an image sensor including a lens array 610 of FIG.6A may more effectively restore consecutive zoom images. Also, a zoomfactor of each lens group may be determined based on a number of lensesincluded in a lens group corresponding to a default zoom (for example,1× zoom). For example, a lens group corresponding to 1× zoom may include“x” lens elements, and a lens group corresponding to z× zoom may include“y” lens elements. In this example, the z× zoom may be y/x times, x maybe an integer greater than or equal to “1” and y may be an integergreater than or equal to “x”. The lens group corresponding to the 1×zoom may be a lens group with a minimum number of lenses among theplurality of lens groups, however, example embodiments are not limitedthereto.

Also, the image sensor including the lens array 610 with the arithmeticprogression structure may effectively restore a high-magnificationimage. For example, when a 4× zoom image is restored, a lens groupcorresponding to 1× zoom may include two lenses on one axis, a lensgroup corresponding to 2× zoom may include four lenses on one axis, anda lens group corresponding to 4× zoom may include eight lenses on oneaxis, in a geometric progression structure. For the same 4× zoom image,in the arithmetic progression structure, a lens group corresponding to1× zoom may include two lenses on one axis, a lens group correspondingto 1.5× zoom may include three lenses on one axis, a lens groupcorresponding to 2× zoom may include four lenses on one axis, a lensgroup corresponding to 2.5× zoom may include five lenses on one axis, alens group corresponding to 3× zoom may include six lenses on one axis,a lens group corresponding to 3.5× zoom may include seven lenses on oneaxis, and a lens group corresponding to 4× zoom may include eight lenseson one axis. In other words, in restoration of the 4× zoom image, animage sensor including a lens array 610 with the geometric progressionstructure may acquire “84” images corresponding to a total number oflenses, and the image sensor including the lens array 610 with thearithmetic progression structure may acquire “203” images. Thus, anamount of information used by the lens array 610 with the arithmeticprogression structure may be 2.42 times that of the geometricprogression structure. Even in sensing information, the lens array 610with the arithmetic progression structure may show a more continuous andgradual change in a density of sensing points, in comparison to the lensarray 610 with the geometric progression structure. Also, the imagesensor including the lens array 610 with the arithmetic progressionstructure may use images corresponding to “9×9” lens elements includedin a lens group corresponding to 4.5× zoom and images (for example,partial images of each image) corresponding to “10×10” lens elementsincluded in a lens group corresponding to 5× zoom, to restore the 4×zoom image.

FIG. 6B illustrates an example of a lens array 610 in which one of aplurality of lens groups includes a single lens.

The lens array 610 includes a first lens group 611 b that includes asingle lens element, and lens groups 612 b through 619 b that eachinclude a plurality of lens elements, and accordingly both a quality ofa wide angle image and a quality of an image corresponding to a zoomfactor may be enhanced.

Also, a number of lenses on one axis for each of the plurality of lensgroups in the lens array 610 may be determined based on a progressionfrom which at least one prime number in an arithmetic progression isexcluded. For example, when a lens group corresponding to 1× zoomincludes a single lens in the lens array 610, a number of lenses on oneaxis in the lens group may be “1”. A single lens-based arithmeticprogression may be 1, 2, 3, 4, 5, 6, 7, 8, 9. In the general arithmeticprogression described above, a least common multiple may be “2520”. Aprogression from which “7” that is a portion of prime numbers in anarithmetic progression is excluded may be 1, 2, 3, 4, 5, 6, 8, 9, 10. Inthe above-described progression from which a portion of the primenumbers is excluded, a least common multiple may be “360”. A number ofsensing elements included in a sensing array may be determined based ona least common multiple of a number of lenses included in lens groupsaccording to Equation 11 described above, and thus the smaller the leastcommon multiple, the easier selection of the number (for example, asensor resolution) of sensing elements included in the sensing array.For example, in the first lens group 611 b through a ninth lens group619 b may include one lens, four lenses, nine lenses, “16” lenses, “25”lenses, “36” lenses, “64” lenses, “81” lenses, and “100” lenses,respectively, on one axis. The lens array 610 of FIG. 6B may providezoom factors of 1×, 2×, 3×, 4×, 5×, 6×, 8×, 9×, and 10×. Although “7” isused as a prime number excluded from an arithmetic progression has beendescribed above, example embodiments are not limited thereto. Forexample, a prime number greater than or equal to “7” may be excluded.

FIG. 6C is a cross-sectional view illustrating the first lens group 611a and the second lens group 612 a of FIG. 6A and a sensing array 620that is disposed below the first lens group 611 a and the second lensgroup 612 a.

Overlapping of sensing points may be minimized by an arrangement of lenselements for each lens group. A lens element included in one lens groupmay represent a field of view (FOV) different from that of a lenselement included in another lens group. A density of sensing points mayvary depending on an area covered by an individual lens group. Also, aparameter of a lens array may be determined such that overlapping ofsensing points in each lens group may be minimized. The parameter of thelens array may include, for example, a gap 682 between lens groups, afocal length 681 of an individual lens group, or a number of sensingelements included in a sensing area 683 covered by a lens group in thesensing array 620.

For example, the lens groups may be spaced apart from each other on aplane corresponding to the lens array. A lens group may be disposed at aposition shifted by a sub-pixel unit from another lens group along avertical axis (for example, a y-axis) and/or a horizontal axis (forexample, an x-axis) on the above-described plane. The sub-pixel unit maybe less than an integer pixel unit. For example, a length of a pitch ofa sensing element may be an integer pixel unit, and the sub-pixel unitmay indicate a length less than the pitch of the sensing element.

Although all the lens elements have the same focal length, for example,the focal length 681, in FIG. 6C, example embodiments are not limitedthereto. For example, a focal length of at least one lens group amonglens groups may be different from a focal length of another lens group.Since a FOV of a lens element included in a lens group changes based onthe focal length 681, overlapping of sensing points may be minimized bya design of the focal length 681.

Also, the sensing area 683 covered by each of all the lens groups hasthe same size and a number of sensing elements covered by each of thelens groups is the same in FIG. 6C, however, example embodiments are notlimited thereto. For example, a number of sensing elements included in asensing area covered by at least one lens group among the lens groupsmay be different from a number of sensing elements covered by anotherlens group.

As described above, the parameter of the lens array, for example, thegap 682, the focal length 681, or a number of sensing elements coveredby the lens groups, may be set such that overlapping of sensing pointsmay be minimized in a design and manufacturing of an image sensor.

FIGS. 7 through 10 are diagrams illustrating examples of an arrangementof lens elements according to an example embodiments.

In a lens array 710 of FIG. 7, one of a plurality of lens groups mayinclude a single lens element. For example, a first lens group 711 mayinclude a single lens element. Each of the other lens groups 712, 713and 714 may include a plurality of lens elements, as described above.Thus, an image sensor may restore a scene image based on sensinginformation sensed through the lens array 710 having a structure of FIG.7 so that both a quality of a wide angle image and a quality of an imagecorresponding to a zoom factor may be enhanced. Based on a structure ofFIG. 7, overlapping of sensing points may also be minimized by anarrangement of lens elements for each lens group.

In a lens array 810 of FIG. 8, lens elements may be arranged based on alens size. For example, one lens element from among a plurality of lenselements may be located closer to a central position of the lens array810 than another lens element that has a lens size greater than a lenssize of the one lens element. Also, one lens element among the pluralityof lens elements may be located farther away from the central positionof the lens array 810 than another lens element that has a lens sizeless than a lens size of the one lens element.

Referring to FIG. 8, first lens elements 811 included in a first lensgroup having a greatest first lens size among four lens groups may belocated on an edge of the lens array 810. Second lens elements 812having a second lens size less than the first lens size may be locatedcloser to the central position than the first lens elements 811. Thirdlens elements 813 having a third lens size less than the second lenssize may be located closer to the central position than the second lenselements 812. Fourth lens elements 814 having a fourth lens size lessthan the third lens size may be located closer to the central positionthan the third lens elements 813.

A lens array of FIG. 9 may include lens elements that are arranged inthe same way as or in a similar way to the lens array 810 of FIG. 8. InFIG. 9, each of a plurality of lens elements may cover the same numberof sensing elements. For example, each of first lens elements 911 maycover 16 sensing elements arranged in a 4×4 shape. Similarly, each ofsecond lens elements 912, third lens elements 913, and fourth lenselements 914 may also cover 16 sensing elements. In this example, a sizeof a sensing element in a sensing array 920 may be designed so that thesame number of sensing elements may be covered by every lens element.For example, a first size (for example, a pixel size) of a first sensingelement covered by a first lens element 911 may be greater than a secondsize of a second sensing element covered by a second lens element 912.Also, a third size of a third sensing element may be less than thesecond size, and a fourth size of a fourth sensing element may be lessthan the third size. In the present disclosure, a size of a sensingelement may be, for example, a pixel pitch of the sensing element.According to an example embodiment, a lens element that provides a smallFOV may cover the same number of sensing elements as that of a lenselement that provides a large FOV. Thus, a degree of a decrease in aresolution due to multiple lenses may be reduced and a quality of ascene image to be restored may be enhanced.

Although a number of sensing elements covered by each of the lenselements is set to an integer as shown in FIG. 9 for convenience ofdescription, example embodiments are not limited thereto. For example,each of the lens elements may cover a fractional number of sensingelements.

In a lens array 1010 of FIG. 10, each of a plurality of lens elements1011 may be randomly arranged with respect to a plurality of sensingelements on a plane corresponding to the lens array 1010. A number oflens elements 1011 may satisfy Equation 9 for each lens size, and thelens elements 1011 may be arranged on a plane of the lens array 1010.Based on a portion of random arrangements of lenses, an image sensor mayrestore a scene image with a high resolution uniformly regardless ofzoom factors supported by an individual lens size as well as a wideangle.

FIG. 11 is a block diagram illustrating a structure of an image sensoraccording to an example embodiment.

Referring to FIG. 11, an image sensor 1100 includes a lens array 1111, asensing array 1120, and a processor 1130.

Referring to FIG. 11, a lens array 1111 may include a plurality of lenselements. The plurality of lens elements may be located on a lens plane.All the plurality of lens elements may have the same or similar focallengths. A plurality of lens elements may be designed to have the samefocal lengths, but the focal lengths may be slightly different from eachother due to a manufacturing tolerance. For example, a differencebetween focal lengths of the plurality of lens elements may be less thana threshold error. As described above, a lens size of at least one ofthe plurality of lens elements may be different from a lens size of atleast one other lens element. Each of the plurality of lens elements mayrefract light and form a focal point at a point on the sensing array1120 including a plurality of sensing elements.

The sensing array 1120 may include a plurality of sensing elements. Theplurality of sensing elements may be located on a sensing plane parallelto a lens plane. The plurality of sensing elements may be located on thesensing plane spaced apart from the lens array 1111 by a distance equalto the focal length of the lens elements. Each of the plurality ofsensing elements may sense light passing through the lens array 1111.For example, each of the plurality of sensing elements may receive lightpassing through a lens element that covers a corresponding sensingelement.

The processor 1130 may restore a scene image based on an intensity oflight sensed by the plurality of sensing elements. For example, theprocessor 1130 may acquire a CEV image based on sensing informationsensed by the plurality of sensing elements. The processor 1130 mayrestore the scene image from the CEV image. The scene image may be animage output from the image sensor 1100, and may be restored to be thesame as or similar to the original scene.

In an example, the processor 1130 may restore a scene image from a CEVimage based on a geometric structure between the plurality of sensingelements and the plurality of lens elements. For example, the processor1130 may rearrange pixels of a captured image (for example, a CEV image)to generate an output image wherein pixels of sensing elements thatsense similar LF information from among the plurality of sensingelements may be rearranged to be adjacent to each other, as describedabove with reference to FIG. 3.

In another example, the processor 1130 may restore a scene image from aCEV image based on an image restoration model that is completely trainedbefore the CEV image is acquired. The image restoration model may be amodel designed to output a scene image corresponding to an arbitrary CEVimage, and may have a machine learning structure. For example, the imagerestoration model may have a neural network structure. The imagerestoration model may be trained to output a reference output imagegiven as a ground truth in response to an input of an arbitraryreference CEV image. However, the image restoration model is not limitedthereto.

FIGS. 12 and 13 are diagrams illustrating examples of an apparatus inwhich an image sensor is implemented according to an example embodiment.

An image sensor may be applicable to various technical fields. The imagesensor may be designed so that a lens array including a plurality oflenses may be spaced apart from a sensor including a plurality ofsensing elements by a relatively small distance equal to the focallength of the lens elements in the lens array. For example, the imagesensor may be implemented as an ultra-thin camera with a large sensorfor high-definition capturing. In other words, a thickness of the imagesensor may be reduced using a multi-lens array structure. The imagesensor may be implemented as an application processor (AP), afield-programmable gate array (FPGA) or a chip, and may be used as animage signal processor of a camera.

Also, the image sensor may acquire sensing information associated with aplurality of zoom factors using a lens array that has an ultra-thinstructure and lens elements that have different lens sizes and the samefocal length. Thus, the image sensor may restore a high resolution sceneimage with respect to the plurality of zoom factors.

In an example, the image sensor may be implemented in a mobile terminal.The mobile terminal may be a terminal movable instead of being fixed atany geographical position, and may include, for example, a portabledevice (for example, a smart device, such as a smartphone or a tabletcomputer), an artificial intelligence speaker, or a vehicle. Examples ofa mobile terminal are shown in FIGS. 12 and 13, however, exampleembodiments are not limited thereto.

Referring to FIG. 12, an image sensor 1210 may be applied to a frontcamera or rear camera of a smartphone. The image sensor 1210 may have astructure in which a large full frame sensor and a micro-lens array arecombined, and may be applied to a camera of a mobile phone. For example,the image sensor 1210 may be included in a front camera or rear cameraof a smart device 1200, as shown in FIG. 12. A sensing array and a lensarray of the image sensor 1210 may be implemented as, for example, afull-frame sensing array and a micro lens array, respectively.

In another example, the image sensor may have a thin structure or acurved structure and may be implemented for vehicles. Referring to FIG.13, an image sensor 1310 may be implemented as a curved front camera orrear camera of a vehicle 1300. However, example embodiments are notlimited thereto, and the image sensor 1310 may be used in, for example,a digital single-lens reflex (DSLR) camera, a drone, a closed-circuittelevision (CCTV), a webcam camera, a panoramic camera, a camera formovies or broadcasts, or a virtual reality/augmented reality (VR/AR)camera. Also, the image sensor 1310 may be applicable to various fields,for example, a flexible/stretchable camera, a compound-eye camera, or acontact lens type camera.

In still another example, the image sensor may also be applicable to amulti-frame super resolution image restoration for increasing aresolution of a video image based on information about a plurality ofconsecutive frames that are captured.

The example embodiments described herein may be implemented usinghardware components, software components, or a combination thereof. Aprocessing device may be implemented using one or more general-purposeor special purpose computers, such as, for example, a processor, acontroller and an arithmetic logic unit, a digital signal processor, amicrocomputer, a field programmable array, a programmable logic unit, amicroprocessor or any other device capable of responding to andexecuting instructions in a defined manner. The processing device mayrun an operating system (OS) and one or more software applications thatrun on the OS. The processing device also may access, store, manipulate,process, and create data in response to execution of the software. Forpurpose of simplicity, a processing device is described as a singleprocessing device; however, one skilled in the art will appreciate thata processing device may include multiple processing elements andmultiple types of processing elements. For example, a processing devicemay include multiple processors or a processor and a controller. Inaddition, different processing configurations are possible, such aparallel processors.

The software may include a computer program, a piece of code, aninstruction, or some combination thereof, to independently orcollectively instruct or configure the processing device to operate asdesired. Software and data may be embodied permanently or temporarily inany type of machine, component, physical or virtual equipment, computerstorage medium or device, or in a propagated signal wave capable ofproviding instructions or data to or being interpreted by the processingdevice. The software also may be distributed over network coupledcomputer systems so that the software is stored and executed in adistributed fashion. The software and data may be stored by one or morenon-transitory computer readable recording media.

The method according to the above-described example embodiments may berecorded in non-transitory computer-readable media including programinstructions to implement various operations which may be performed by acomputer. The media may also include, alone or in combination with theprogram instructions, data files, data structures, and the like. Theprogram instructions recorded on the media may be those speciallydesigned and constructed for the purposes of the example embodiments, orthey may be of the well-known kind and available to those having skillin the computer software arts. Examples of non-transitorycomputer-readable media include magnetic media such as hard disks,floppy disks, and magnetic tape; optical media such as CD ROM discs andDVDs; magneto-optical media such as optical discs; and hardware devicesthat are specially configured to store and perform program instructions,such as read-only memory (ROM), random access memory (RAM), flashmemory, and the like. Examples of program instructions include bothmachine code, such as code produced by a compiler, and files containinghigher level code that may be executed by the computer using aninterpreter. The described hardware devices may be configured to act asone or more software modules in order to perform the operations of theabove-described example embodiments, or vice versa.

While this disclosure includes example embodiments, it will be apparentto one of ordinary skill in the art that various changes in form anddetails may be made in these example embodiments without departing fromthe spirit and scope of the claims and their equivalents. The exampleembodiments described herein are to be considered in a descriptive senseonly, and not for purposes of limitation. Descriptions of features oraspects in each example are to be considered as being applicable tosimilar features or aspects in other examples. Suitable results may beachieved if the described techniques are performed in a different order,and/or if components in a described system, architecture, device, orcircuit are combined in a different manner and/or replaced orsupplemented by other components or their equivalents. Therefore, thescope of the disclosure is defined not by the detailed description, butby the claims and their equivalents, and all variations within the scopeof the claims and their equivalents are to be construed as beingincluded in the disclosure.

What is claimed is:
 1. An image sensor comprising: a lens arraycomprising a plurality of lens groups, each of the plurality of lensgroups comprising at least one lens element; and a sensing arraycomprising a plurality of sensing elements spaced apart from the lensarray and configured to sense light passing through the lens array,wherein a lens size of one lens group from among the plurality of lensgroups is different from a lens size of another lens group from amongthe plurality of lens groups, wherein a number of sensing elementsarranged along one axis of the sensing array is determined based on anumber of lenses for each lens group of the plurality of lens groups,wherein the number of lenses of each lens group of the plurality of lensgroups is different from the number of lenses of each other lens groupof the plurality of lens groups, and wherein the number of sensingelements arranged along the one axis of the sensing array is determinedbased on a least common multiple of the number of lenses for each of theplurality of lens groups.
 2. The image sensor of claim 1, wherein thenumber of lenses for each of the plurality of lens groups is determinedbased on a geometric progression.
 3. The image sensor of claim 1,wherein the number of lenses for each of the plurality of lens groups isdetermined based on an arithmetic progression.
 4. The image sensor ofclaim 1, wherein the number of lenses for each of the plurality of lensgroups is determined based on a progression from which at least oneprime number in an arithmetic progression is excluded.
 5. The imagesensor of claim 1, wherein the number of lenses for each of theplurality of lens groups is determined based on a progression that isbased on a number of lenses included in a lens group corresponding to adefault zoom.
 6. The image sensor of claim 1, wherein each lens groupfrom among the plurality of lens groups corresponds to a respective lenssize such that for each lens group from among the plurality of lensgroups, lens elements included in the lens group have the correspondinglens size.
 7. The image sensor of claim 1, wherein for each lens groupfrom among the plurality of lens groups, lens elements included in thelens group are located adjacent to each other.
 8. The image sensor ofclaim 1, wherein one lens group from among the plurality of lens groupscomprises a single lens element.
 9. The image sensor of claim 1, whereinthe at least one lens element is arranged to cover less than an entireportion of at least one sensing element from among the plurality ofsensing elements.
 10. The image sensor of claim 1, further comprising: aprocessor configured to restore an image based on sensing informationsensed by the plurality of sensing elements so that a resolution of acentral region within a field of view (FOV) of the lens array is higherthan a resolution of a region adjacent to the central region.
 11. Theimage sensor of claim 1, further comprising: a processor configured toacquire a compound eye vision (CEV) image based on sensing informationsensed by the plurality of sensing elements.
 12. The image sensor ofclaim 11, wherein the processor is further configured to rearrangepixels included in the CEV image based on light field (LF) informationsensed by the plurality of sensing elements.
 13. The image sensor ofclaim 11, wherein the processor is further configured to restore a sceneimage from the CEV image based on a geometric relationship between theplurality of sensing elements and the plurality of lens groups.
 14. Theimage sensor of claim 11, wherein the processor is further configured torestore a scene image from the CEV image based on an image restorationmodel that is completely trained before the CEV image is acquired. 15.The image sensor of claim 1, further comprising: a processor configuredto select target sensing information from among sensing informationsensed by the plurality of sensing elements, the target sensinginformation corresponding to a zoom factor designated by a user, whereinthe processor is further configured to restore a scene image based onthe selected target sensing information.
 16. The image sensor of claim15, wherein the processor is further configured to select, as the targetsensing information, information corresponding to a field of viewcorresponding to the designated zoom factor.
 17. The image sensor ofclaim 1, wherein each lens element from among the plurality of lensgroups is configured to refract incident light and to form a focal pointof light exiting the lens element at a point on the sensing arraycomprising the plurality of sensing elements.
 18. An image sensingmethod comprising: sensing, by a sensing array comprising a plurality ofsensing elements, light passing through a lens array comprising aplurality of lens groups, each comprising at least one lens element; andrestoring, by a processor, a scene image based on an intensity of thelight sensed by the sensing array, wherein a lens size of one lens groupfrom among the plurality of lens groups is different from a lens size ofanother lens group from among the plurality of lens groups, wherein anumber of sensing elements arranged along one axis of the sensing arrayis determined based on a number of lenses for each of the plurality oflens groups, wherein the number of lenses of each lens group of theplurality of lens groups is different from the number of lenses of eachother lens group of the plurality of lens groups, and wherein the numberof sensing elements arranged along the one axis of the sensing array isdetermined based on a least common multiple of the number of lenses foreach of the plurality of lens groups.
 19. A camera comprising: a lensarray comprising a plurality of lens groups, each of the plurality oflens groups comprising at least one lens element; and a sensing arraycomprising a plurality of sensing elements spaced apart from the lensarray and configured to sense light passing through the lens array,wherein a lens size of one lens group from among the plurality of lensgroups is different from a lens size of another lens group from amongthe plurality of lens groups, wherein a number of sensing elementsarranged along one axis of the sensing array is determined based on anumber of lenses for each of the plurality of lens groups, wherein thenumber of lenses of each lens group of the plurality of lens groups isdifferent from the number of lenses of each other lens group of theplurality of lens groups, and wherein the number of sensing elementsarranged along the one axis of the sensing array is determined based ona least common multiple of the number of lenses for each of theplurality of lens groups.